Skip to content

upgrade pinot lib to support new features introduced in Apache Pinot 0.9.3#17015

Merged
highker merged 1 commit intoprestodb:masterfrom
xiangfu0:update_pom_to_0.9.0
Feb 3, 2022
Merged

upgrade pinot lib to support new features introduced in Apache Pinot 0.9.3#17015
highker merged 1 commit intoprestodb:masterfrom
xiangfu0:update_pom_to_0.9.0

Conversation

@xiangfu0
Copy link
Copy Markdown
Contributor

@xiangfu0 xiangfu0 commented Nov 18, 2021 •
edited by highker
Loading

Test plan - (Please fill in how you tested your changes)

Please make sure your submission complies with our Development, Formatting, and Commit Message guidelines. Don't forget to follow our attribution guidelines for any code copied from other projects.

Fill in the release notes towards the bottom of the PR description.
See Release Notes Guidelines for details.

== RELEASE NOTES ==

Pinot Changes
* Upgrade to support Pinot version 0.9.3
* Support Pinot `TIMESTAMP` and `JSON` types

@xiangfu0 xiangfu0 marked this pull request as draft November 18, 2021 00:59
@xiangfu0 xiangfu0 force-pushed the update_pom_to_0.9.0 branch 3 times, most recently from 4445ee1 to 5c427ce Compare November 19, 2021 20:01
@Eywek
Copy link
Copy Markdown
Contributor

Eywek commented Jan 24, 2022

Hello,

Do you know when you'll be able to end this PR?
I'm currently using Pinot 0.9 and I'm unable to use Presto (missing JSON columns...)

Thank you

@xiangfu0
Copy link
Copy Markdown
Contributor Author

xiangfu0 commented Jan 24, 2022

Hello,

Do you know when you'll be able to end this PR? I'm currently using Pinot 0.9 and I'm unable to use Presto (missing JSON columns...)

Thank you

I'm still working on testing this feature, will update once I finished this.

@Eywek
Copy link
Copy Markdown
Contributor

Eywek commented Jan 24, 2022

Great! Thank you for the update
Do you have any idea when (approximately) you will be able to end this?

@xiangfu0 xiangfu0 force-pushed the update_pom_to_0.9.0 branch from 5c427ce to 72c8934 Compare January 24, 2022 22:03
@xiangfu0 xiangfu0 changed the title upgrade pinot lib to support new features introduced in Apache Pinot 0.9.0 upgrade pinot lib to support new features introduced in Apache Pinot 0.9.3 Jan 24, 2022
@xiangfu0 xiangfu0 force-pushed the update_pom_to_0.9.0 branch from 72c8934 to 1dce244 Compare January 24, 2022 22:18
@xiangfu0
Copy link
Copy Markdown
Contributor Author

xiangfu0 commented Jan 24, 2022

Great! Thank you for the update Do you have any idea when (approximately) you will be able to end this?

hopefully this week

@xiangfu0 xiangfu0 marked this pull request as ready for review January 25, 2022 00:09
@xiangfu0 xiangfu0 marked this pull request as draft January 25, 2022 00:10
@xiangfu0 xiangfu0 force-pushed the update_pom_to_0.9.0 branch 6 times, most recently from 31c644e to d61767d Compare January 25, 2022 07:05
@xiangfu0 xiangfu0 closed this Jan 25, 2022
@xiangfu0 xiangfu0 reopened this Jan 25, 2022
@xiangfu0 xiangfu0 force-pushed the update_pom_to_0.9.0 branch 8 times, most recently from e5f7ef4 to e698b11 Compare January 26, 2022 08:55
@xiangfu0
Copy link
Copy Markdown
Contributor Author

xiangfu0 commented Jan 26, 2022

Great! Thank you for the update Do you have any idea when (approximately) you will be able to end this?

I've tested this locally with default mode and grpc mode, you can grab the code and test it as well.

@xiangfu0
Copy link
Copy Markdown
Contributor Author

xiangfu0 commented Jan 26, 2022

@dharakk please take a look once you got time. This is for pinot 0.9.3 upgrade, should be compatible with older pinot version, but better to test it with your own environment

@xiangfu0 xiangfu0 force-pushed the update_pom_to_0.9.0 branch from e698b11 to 6685ea4 Compare January 26, 2022 22:10
@xiangfu0 xiangfu0 force-pushed the update_pom_to_0.9.0 branch from 6685ea4 to 9536e59 Compare January 26, 2022 23:35
@xiangfu0 xiangfu0 marked this pull request as ready for review January 27, 2022 08:34
@Eywek
Copy link
Copy Markdown
Contributor

Eywek commented Jan 27, 2022

I've tested this locally with default mode and grpc mode, you can grab the code and test it as well.

Thank you, I'll try in the day. I've already tested 2 days ago (so before last changes) and everything seemed to work except when I had a split that have 0 rows. Pinot doesn't return the data schema which leads to a Presto crash (i.e. when trying to call dataTable.getDataSchema().size() since dataSchema = null). I'll try again with all changes and see if this is fixed

@xiangfu0
Copy link
Copy Markdown
Contributor Author

xiangfu0 commented Jan 27, 2022

I've tested this locally with default mode and grpc mode, you can grab the code and test it as well.

Thank you, I'll try in the day. I've already tested 2 days ago (so before last changes) and everything seemed to work except when I had a split that have 0 rows. Pinot doesn't return the data schema which leads to a Presto crash (i.e. when trying to call dataTable.getDataSchema().size() since dataSchema = null). I'll try again with all changes and see if this is fixed

Sure, I assume you are using pinot server grpc endpoint right? The last block is metadata block, I've fixed that.

Please give it a try. Let me know if there is any issue.

@Eywek
Copy link
Copy Markdown
Contributor

Eywek commented Jan 31, 2022

Please give it a try. Let me know if there is any issue.

Lgtm, issues I had are fixed

@xiangfu0
Copy link
Copy Markdown
Contributor Author

xiangfu0 commented Jan 31, 2022

Please give it a try. Let me know if there is any issue.

Lgtm, issues I had are fixed

Thanks for your validation!

@xiangfu0
Copy link
Copy Markdown
Contributor Author

xiangfu0 commented Feb 1, 2022

@highker please take a look once you got time! Thanks!

@highker highker self-requested a review February 1, 2022 05:43
@highker highker merged commit efd857f into prestodb:master Feb 3, 2022
@neeradsomanchi neeradsomanchi mentioned this pull request Feb 8, 2022
Merged
4 tasks
@xiangfu0 xiangfu0 deleted the update_pom_to_0.9.0 branch March 15, 2022 18:48
@pratyakshsharma
Copy link
Copy Markdown
Contributor

pratyakshsharma commented Jun 8, 2022

this is to bring to your notice that I am not able to connect to any catalog when I run presto on my intellij on my M1 mac. This has been happening since I pulled this commit in my local machine. Errors like "bad file descriptor", "Could not create connection to database server. Attempted reconnect 3 times. Giving up" have been coming when I try to connect to any catalog from the cli on my laptop. @highker @xiangfu0
Also this error does not come if I try to run the server using tar file.
I am attaching a sample stacktrace below -

com.facebook.presto.spi.PrestoException: The connection attempt failed.
at com.facebook.presto.plugin.jdbc.BaseJdbcClient.getSchemaNames(BaseJdbcClient.java:157)
at com.facebook.presto.plugin.jdbc.JdbcMetadata.listSchemaNames(JdbcMetadata.java:75)
at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorMetadata.listSchemaNames(ClassLoaderSafeConnectorMetadata.java:212)
at com.facebook.presto.metadata.MetadataManager.listSchemaNames(MetadataManager.java:311)
at com.facebook.presto.connector.informationSchema.InformationSchemaMetadata.calculatePrefixesWithSchemaName(InformationSchemaMetadata.java:276)
at com.facebook.presto.connector.informationSchema.InformationSchemaMetadata.getTableLayouts(InformationSchemaMetadata.java:240)
at com.facebook.presto.metadata.MetadataManager.getLayout(MetadataManager.java:396)
at com.facebook.presto.sql.planner.iterative.rule.PickTableLayout$PickTableLayoutWithoutPredicate.apply(PickTableLayout.java:211)
at com.facebook.presto.sql.planner.iterative.rule.PickTableLayout$PickTableLayoutWithoutPredicate.apply(PickTableLayout.java:178)
at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.transform(IterativeOptimizer.java:168)
at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreNode(IterativeOptimizer.java:141)
at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:106)
at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreChildren(IterativeOptimizer.java:191)
at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:108)
at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreChildren(IterativeOptimizer.java:191)
at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:108)
at com.facebook.presto.sql.planner.iterative.IterativeOptimizer.optimize(IterativeOptimizer.java:95)
at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:195)
at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:183)
at com.facebook.presto.execution.SqlQueryExecution.lambda$doAnalyzeQuery$2(SqlQueryExecution.java:446)
at com.facebook.presto.common.RuntimeStats.profileNanos(RuntimeStats.java:129)
at com.facebook.presto.execution.SqlQueryExecution.doAnalyzeQuery(SqlQueryExecution.java:444)
at com.facebook.presto.execution.SqlQueryExecution.analyzeQuery(SqlQueryExecution.java:430)
at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:366)
at com.facebook.presto.$gen.Presto_null__testversion____20220607_111344_1.run(Unknown Source)
at com.facebook.presto.execution.SqlQueryManager.createQuery(SqlQueryManager.java:266)
at com.facebook.presto.dispatcher.LocalDispatchQuery.lambda$startExecution$8(LocalDispatchQuery.java:197)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:292)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:454)
at org.postgresql.Driver.connect(Driver.java:256)
at com.facebook.presto.plugin.jdbc.DriverConnectionFactory.openConnection(DriverConnectionFactory.java:82)
at com.facebook.presto.plugin.jdbc.BaseJdbcClient.getSchemaNames(BaseJdbcClient.java:151)
... 29 more
Caused by: java.net.SocketException: Bad file descriptor (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.postgresql.core.PGStream.flush(PGStream.java:527)
at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:402)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:94)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192)
... 35 more

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@highker highker highker approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@xiangfu0 @Eywek @pratyakshsharma @highker